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Abstract. We develop a discrete analogue of Hamilton-Jacobi theory in the framework of discrete 
Hamiltonian mechanics. The resulting discrete Hamilton-Jacobi equation is discrete only in time. 
We describe a discrete analogue of Jacobi's solution and also prove a discrete version of the geometric 
Hamilton-Jacobi theorem. The theory applied to discrete linear Hamiltonian systems yields the 
y—{ discrete Riccati equation as a special case of the discrete Hamilton-Jacobi equation. We also apply 

t-H the theory to discrete optimal control problems, and recover some well-known results, such as 

the Bellman equation (discrete-time HJB equation) of dynamic programming and its relation to 
the costate variable in the Pontryagin maximum principle. This relationship between the discrete 
Hamilton-Jacobi equation and Bellman equation is exploited to derive a generalized form of the 
Bellman equation that has controls at internal stages. 

oo 

1. Introduction 

1.1. Discrete Mechanics. Discrete mechanics is a reformulation of Lagrangian and Hamiltonian 
mechanics with discrete time, as opposed to a discretization of the equations in the continuous- 
time theory. It not only provides a systematic view of structure-preserving integrators, but also 
has interesting theoretical aspects analogous to continuous-time Lagrangian and Hamiltonian me- 
chanics [see, e.g.. I30 ( 1331 134] . The main feature of discrete mechanics is its use of discrete versions 

CNI of variational principles. Namely, discrete mechanics assumes that the dynamics is defined at dis- 

crete times from the outset, formulates a discrete variational principle for such dynamics, and then 
derives a discrete analogue of the Euler-Lagrange or Hamilton's equations from it. 

The advantage of this construction is that it naturally gives rise to discrete analogues of the 
concepts and ideas in continuous time that have the same or similar properties, such as symplectic 
t— l forms, the Legendre transformation, momentum maps, and Noether's theorem [30j. This in turn 

provides us with the discrete ingredients that facilitate further theoretical developments, such as 
discrete analogues of the theories of complete integrability [see, e.g., [31j (33j [M] and also those 
of reduction and connections |20| [25j 128] . Whereas the main topic in discrete mechanics is the 
development of structure-preserving algorithms for Lagrangian and Hamiltonian systems [see, e.g., 
130] . the theoretical aspects of it are interesting in their own right, and furthermore provide insight 
into the numerical aspects as well. 

Another notable feature of discrete mechanics, especially on the Hamiltonian side, is that it is 
a generalization of (nonsingular) discrete optimal control problems. In fact, as stated in Marsden 
and West [30], discrete mechanics is inspired by discrete formulations of optimal control problems 
(see, e.g., Jordan and Polak [21J and Cadzow [9]). 

1.2. Hamilton-Jacobi Theory. In classical mechanics [see, e.g.. 13} IT6 l 124 } [29]. the Hamilton- 
Jacobi equation is first introduced as a partial differential equation that the action integral satisfies. 
Specifically, let Q be a configuration space and T*Q be its cotangent bundle; and let q G Q and 
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t > be arbitrary and suppose that (q(s),p(s)) G T*Q is a solution of Hamilton's equations 

dH . _ _ dH 

dp ' ^ dq 



(1.1) 



with the endpoint condition q(t) = q. Then calculate the action integral along the solution over 
the time interval [0,t], i.e., 



S(q,t) := / \p(s) ■ q(s) — H 
Jo L 



(q(s),p(s)) 



ds, 



(1.2) 



where we regard the resulting integral as a function of the endpoint (q, t) G Qx IR + with being 
the set of positive real numbers. Then by taking variation of the endpoint (q,t), one obtains a 
partial differential equation satisfied by S(q,t): 



dS 
~di 



+ H[q, 



as 

dq 



0. 



;i.3) 



This is the Hamilton- Jacobi equation. 

Conversely, it is shown that if S(q, t) is a solution of the Hamilton-Jacobi equation then S(q, t) is 
a generating function for the family of canonical transformations (or symplectic flows) that describe 
the dynamics defined by Hamilton's equations. This result is the theoretical basis for the powerful 
technique of exact integration called separation of variables. 

1.3. Connection with Optimal Control and The Hamilton Jacobi Bellman Equation. 

The idea of Hamilton-Jacobi theory is also useful in optimal control theory (see, e.g., Jurdjevic 
|22j and Bertsekas [6J). Consider a typical optimal control problem 

cT 

C(q, u) dt, 



mm 

«(■) Jo 



subject to the constraints, 

Q = f(q,u), 

and q(0) = qo and q(T) = qx- We define the augmented cost functional: 



S\u] 



p-q- H(q,p, u) 



{C(q,u)+p[q- f(q,u)}}dt = 

where we introduced the costate p, and also defined the control Hamiltonian, 

H(q,p, u):=p- f(q, u) - C(q, u). 

Assuming that 



dt 



8H_ 

du 



(q,p,u) = 



uniquely defines the optimal control u = u*(q,p), we set 

H(q,p) := maxH(q,p,u) = H(q,p,u* (q,p)) . 

u 

We also define the optimal cost-to-go function 



J(q, t) := J t T {c(q, U*) + p[q - f(q, U*)) } ds 



T r 



p-q-H(q,p) ds = S* - S(q,t), 
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where (q(s),p(s)) for s £ [0, T] is the solution of Hamilton's equations with the above H such that 
q(t) = q; and S* is the optimal cost 



S* 



p-q- H(q,p) 



ds 



p- q- H(q,p,u*(q,p)) ds = S[u*], 



and the function S(q,t) is defined by 

S(q,t) :- 



t r 



p-q- H(q,p) 



ds. 



Since this definition coincides with Eq. (1.2), the function S(q,t) = S* — J(q,t) satisfies the H- 
J equation (1.3); this reduces to the Hamilton-Jacobi-Bellman (HJB) equation for the optimal 
cost-to-go function J(q,t): 



dJ_ 

at 



+ min 



dj_ 

dq 



f(q,u) + C(q,u) 



0. 



(1.4) 



It can also be shown that the costate p of the optimal solution is related to the solution of the HJB 
equation. 

1.4. Discrete Hamilton Jacobi Theory. The main objective of this paper is to present a dis- 
crete analogue of Hamilton-Jacobi theory within the framework of discrete Hamiltonian mechan- 
ics [23], and also to apply the theory to discrete optimal control problems. 

There are some previous works on discrete-time analogues of the Hamilton-Jacobi equation, 
such as Elnatanov and Schiff [13] and Lall and West [23]. Specifically, Elnatanov and Schiff |13j 
derived an equation for a generating function of a coordinate transformation that trivializes the 
dynamics. This derivation is a discrete analogue of the conventional derivation of the continuous- 
time Hamilton-Jacobi equation [see, e.g., [Ml Chapter VIII]. Lall and West [23J formulated a discrete 
Lagrangian analogue of the Hamilton-Jacobi equation as a separable optimization problem. 

1.5. Main Results. Our work was inspired by the result of Elnatanov and Schiff [13] and starts 
from a reinterpretation of their result in the language of discrete mechanics. This paper further 
extends the result by developing discrete analogues of results in (continuous-time) Hamilton-Jacobi 
theory. Namely, we formulate a discrete analogue of Jacobi's solution, which relates the discrete 
action sum to a solution of the discrete Hamilton-Jacobi equation. This also provides a very simple 
derivation of the discrete Hamilton-Jacobi equation and exhibits a natural correspondence with 
the continuous-time theory. Another important result in this paper is a discrete analogue of the 
Hamilton-Jacobi theorem, which relates the solution of the discrete Hamilton-Jacobi equation with 
the solution of the discrete Hamilton's equations. 

We also show that the discrete Hamilton-Jacobi equation is a generalization of the discrete 
Riccati equation and the Bellman equation (see Fig. [I]). Specifically, we show that the discrete 
Hamilton-Jacobi equation applied to linear discrete Hamiltonian systems and discrete optimal 
control problems reduces to the discrete Riccati and Bellman equations, respectively. This is again 
a discrete analogue of the well-known results that the Hamilton-Jacobi equation applied to linear 
Hamiltonian systems and optimal control problems reduces to the Riccati (see, e.g., Jurdjevic |22l 
p. 421]) and HJB equations (see Section 1.3 above), respectively. 

The link between the discrete Hamilton-Jacobi equation and the Bellman equation turns out to 
be useful in deriving a class of generalized Bellman equations that are higher-order approximations 
of the original continuous-time problem. Specifically, we use the idea of the Galerkin Hamil- 
tonian variational integrator of Leok and Zhang [26] to derive discrete control Hamiltonians that 
yield higher-order approximations, and then show that the corresponding discrete Hamilton-Jacobi 
equation gives a class of Bellman equations with controls at internal stages. 
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Figure 1. Discrete evolution equations (left) and corresponding discrete Hamilton- 
Jacobi-type equations (right). Dashed lines are the links established in the paper. 



1.6. Outline of the Paper. We first present a brief review of discrete Lagrangian and Hamiltonian 
mechanics in Section [2] In Section [3j we describe a reinterpretation of the result of Elnatanov and 
Schiff [13] in the language of discrete mechanics and a discrete analogue of Jacobi's solution to 
the discrete Hamilton-Jacobi equation. The remainder of Section [3] is devoted to more detailed 
studies of the discrete Hamilton-Jacobi equation: its left and right variants, more explicit forms 
of them, and also a digression on the Lagrangian side. In Section |4j we prove a discrete version 
of the Hamilton-Jacobi theorem. In Section [5j we apply the theory to linear discrete Hamiltonian 
systems, and show that the discrete Riccati equation follows from the discrete Hamilton-Jacobi 
equation. Section [6] establishes the link with discrete-time optimal control and interprets the results 
of the preceding sections in this setting. Section [7] further extends this idea to derive a class of 
Bellman equations with controls at internal stages. 



2. Discrete Mechanics 

This section briefly reviews some key results of discrete mechanics following Marsden and West 
[30] and Lall and West [23]. 

2.1. Discrete Lagrangian Mechanics. A discrete Lagrangian flow {qk}k=oi on an ^-dimensional 
differentiable manifold Q, can be described by the following discrete variational principle: Let 
be the following action sum of the discrete Lagrangian L d : Q x Q — > R: 

w-i r t N 
S?({qk}%=o) ■= E L ^qk+i) « / L(q(t),q(t))dt, (2.1) 
fc=o Jo 

which is an approximation of the action integral as shown above. 

Consider discrete variations qk i-> qk + £ 5qk, for k = 0, 1, . . . , JV, with 5qo = 5qjy = 0. Then, the 
discrete variational principle 5S^ = gives the discrete Euler-Lagrange equations: 

D 2 L d (q k ^ 1 ,q k ) + D 1 L d (q k , q k +i) = 0. (2.2) 

This determines the discrete flow Fi d : Q x Q — > Q x Q: 

Fl a ■ (%-!,%) i-> (qk,qk+i)- (2.3) 
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Let us define the discrete Lagrangian symplectic one-forms 

&t ■ Q xQ^T*(Q x Q) by 
@L d : (lk,qk+i) i-> D 2 L d (q k ,q k+ i)dq k+ i, (2.4a) 
©L d : (ik, Qk+i) H> -DiL d (g fc , q k+1 ) dq k . (2.4b) 
Then, the discrete flow F^ d preserves the discrete Lagrangian symplectic form 

^L d (<?fc,<7fc+i) : =d@t d =d ®I d = D i D 2L d {qk,qk+i)dq k A dq k+1 . (2.5) 
Specifically, we have 

(F Ld yn Ld = n Ld . 

2.2. Discrete Hamiltonian Mechanics. Introduce the right and left discrete Legendre trans- 
forms FL d :QxQ^ T*Q by 

FL^ : (q k ,q k+ i) i-> (q k+ i, D 2 L d (q k , q k+1 )), (2.6a) 

FL d : gjt+i) >-> (g*, -D 1 L d (q k , q k +i)), (2.6b) 

respectively. Then we find that the discrete Lagrangian symplectic forms Eq. (2.4) and (2.5) are 
pull-backs by these maps of the standard symplectic form on T*Q: 

e± = (fl±)*g, n± d = (FL±yn. 

Let us define the momenta 



Pk,k+i := - D i L d(Qk,qk+i), P k , k+ i '■= D 2 L d (q k ,q k+ 



Then, the discrete Euler-Lagrange equations (2.2) become simply p k-1 k = P k k+ i- So defining 

Pk '■= P k -i,k = Pk,k+v 



(2.7) 



one can rewrite the discrete Euler-Lagrange equations (2.2) as follows: 

Pk = -DiL d (q k ,q k+ i), 

Pk+i = D 2 L d {q k ,q k+ i). 
Furthermore, define the discrete Hamiltonian map Fl a ■ T*Q — > T*Q by 

Fh d ■ (qk,Pk) i-> (q k+ i,p k +i). (2.8) 
Then, one may relate this map with the discrete Legendre transforms in Eq. (2.6) as follows: 

F Ld =FL+o(FL d )- 1 . (2.9) 
Furthermore, one can also show that this map is symplectic, i.e., 

(F Ld yn = n. 

This is the Hamiltonian description of the dynamics defined by the discrete Euler-Lagrange equa- 
tion (2.2) introduced by Marsden and West [30J. Notice, however, that no discrete analogue of 
Hamilton's equations is introduced here, although the flow is now on the cotangent bundle T*Q. 

Lall and West [23] pushed this idea further to give discrete analogues of Hamilton's equations: 
From the point of view that a discrete Lagrangian is essentially a generating function of type 
one |16] . we can apply Legendre transforms to the discrete Lagrangian to find the corresponding 
generating functions of type two or three [16] . In fact, they turn out to be a natural Hamiltonian 
counterpart to the discrete Lagrangian mechanics described above. Specifically, with the right 
discrete Legendre transform 

p k+1 =¥L^(q k ,q k+1 ) = D 2 L d (q k ,q k+1 ), (2.10) 
we can define the following right discrete Hamiltonian: 

H d (q k ,p k+ i) = p k+ i ■ q k+ i - L d (q k , q k+1 ). (2.11) 
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Then, the discrete Hamiltonian map Fi d : (qk,Pk) l— (<Zfc+iiPfc+i) is defined implicitly by the right 
discrete Hamilton's equations 

Qk+i = D 2 H+(q k ,p k+1 ), (2.12a) 
Pk = D 1 H+(q k , Pk+1 ), (2.12b) 

which are precisely the characterization of a symplectic map in terms of a generating function, Ht, 
of type two. Similarly, with the left discrete Legendre transform 

Pk = ^L~(q k ,q k+1 ) = -DiL d (q k ,q k+ i), (2.13) 
we can define the following left discrete Hamiltonian: 

H^{pk,qk+i) = ~Pk ■ qk ~ L d (q k ,q k+1 ). (2.14) 
Then, we have the left discrete Hamilton's equations 

qk = -DiH^(p k ,q k+ i), (2.15a) 

Pk+1 = -D 2 H~(p k ,q k+1 ), (2.15b) 

which corresponds to a symplectic map expressed in terms of a generation function, H^ , of type 
three. 

On the other hand, Leok and Zhang [26] demonstrate that discrete Hamiltonian mechanics can 
be obtained as a direct variational discretization of continuous Hamiltonian mechanics, instead of 
having to go via discrete Lagrangian mechanics. 

3. Discrete Hamilton- Jacobi Equation 

3.1. Derivation by Elnatanov and Schiff. Elnatanov and Schiff [13J derived a discrete Hamilton- 
Jacobi equation based on the idea that the Hamilton-Jacobi equation is an equation for a symplectic 
change of coordinates under which the dynamics becomes trivial. In this section, we would like to 
reinterpret their derivation in the framework of discrete Hamiltonian mechanics reviewed above. 

Theorem 3.1. Suppose that the discrete dynamics {(qk,Pk)} k= o is governed by the right discrete 



Hamilton's equations (2.12). Consider the symplectic coordinate transformation (q k ,p k ) •— > {qkiPk) 
that satisfies the following: 

(i) The old and new coordinates are related by the type-one generating functioi^S k : 



p k = -D x S k (q k ,q k ), 
Pk = D 2 S {q k ,q k ); 

(ii) the dynamics in the new coordinates {{qkiPk)} k= o is rendered trivial, i.e., (q k+ i,p k+ i) = 
{qk,Pk)- 

Then, the set of functions {S k }^ =1 satisfies the discrete Hamilton-Jacobi equation: 

S k+1 (qo,q k+ l) ~ S k (q ,q k ) - D 2 S k+1 (q , q k+l ) ■ q k+1 + H+ (q k , D 2 S k+ \q , q k+ i)) = 0, (3.2) 
or, with the shorthand notation S d (q k ) := S k (qo,q k ), 

S k+1 (q k+1 ) - S k (q k ) - DS k+1 (qk+i) • q k +i + H+(q k , DS k+ \q k+l )) = 0. (3.3) 



^This is essentially the same as Eq. (2.71 in the sense that they are both transformations denned by generating func- 



tions of type one: Replace (qh,Pk, Qk+i,Pk+i, Ld) by (qk,Pk,Qk,Pk, S ). However they have different interpretations: 



Eq. (2.71 describes the dynamics or time evolution whereas Eq. (3.1 1 is a change of coordinates 
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Proof. The key ingredient in the proof is the right discrete Hamiltonian in the new coordinates, 
i.e., a function Hf(q k ,p k +i) that satisfies 



or equivalently, 



q k +i = D 2 H^(q k ,Pk+i), 

+ q k+ i dpk+i = dH^(q k ,p k+ i). 



(3.4) 



(3.5) 



Let us first write H f in terms of the original right discrete Hamiltonian Hf and the generating 
function S . For that purpose, first rewrite Eqs. (2.12) and (3.1) as follows: 



Pk dq k = -qk+i dpk+i + dH^(q k ,p k+1 ) 



and 



Pk dq k = Pk dq k - dS k (q k , q k ), 
respectively. Then, using the above relations, we have 

Pk dq k + q k+ i dp k+1 = p k dq k + d{p k+1 ■ q k+1 ) - p k+1 dq k+i 

= Pk dq k - dS k (q k , q k ) + d{p k+1 ■ q k+1 ) - p k+1 dq k+1 + dS k+1 (q k+1 , q k+1 ) 
= -Qk+i dpk+i + dH+(q k ,p k+1 ) 

- dS k (q k ,q k ) + d(p k+1 q k+ i) -p k+1 dq k+1 + dS k+1 (q k +i, q k +i) 
= d(H+(q k ,p k+1 ) +p k+ i ■ q k+ i - p k+ i ■ q k +i + S k+1 (q k+1 ,q k+1 ) - S k (q k ,q k ) \ . 



Qk) 



Thus, in view of Eq. (3.5), we obtain 

H^(qk,Pk+i) = H+(q k ,p k+1 ) + p k+ \ ■ q k+ i - p k+ i ■ q k +i + S k+1 (q k+ i, q k+1 ) - S k (q k , 



(3.6) 



Now consider the choice of the new right discrete Hamiltonian Ht that renders the dynamics 
trivial, i.e., (q k+ i,p k+ i) = (q k ,p k )- It is clear from Eq. (3.4) that we can set 

H^(q k ,Pk+i) = Pk+i • Qk- 



Then, Eq. (3.6) becomes 



Pk+i ■ qk = Hj(q k ,p k+1 ) +p k+ \ ■ q k+ i - Pk+i ■ Qk+i + S + (q k+1 ,q k+1 ) - S (q k ,q k ), 
and since q k +i = q k = ■■■ = %, we have 

= H+(q k ,p k+ i) -p k+ i • q k+ i + 5 fe+1 (g ,^+i) - S k {q ,q k ). 



Eliminating p^+i by using Eq. (3.1), we obtain Eq. (|3.2[). 



□ 



Remark 3.2. What Elnatanov and Schiff [13] refer to as the Hamilton- J acobi difference equation is 
the following: 



(%,qk 



+i, 



S k (q ,q k ) - D 2 S k+1 (q ,q k +i) • D 2 H+ (q k ,p k+1 ) + H+{q k ,p k+1 ) = 0. (3.7) 



It is clear that this is equivalent to Eq. rt3.2| in view of Eq. (2.12) 
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3.2. Discrete Analogue of Jacobi's Solution. This section presents a discrete analogue of 
Jacobi's solution. This also gives an alternative derivation of the discrete Hamilton-Jacobi equation 
that is much simpler than the one shown above. 



Theorem 3.3. Consider the action sums, Eq. (2.1), written in terms of the right discrete Hamil- 
tonian, Eq. (2.11): 



fe-l 



:= J2i Pl+1 ' qi+1 ~ H d(^l,Pl+l) 



(3i 



1=0 



evaluated along a solution of the right discrete Hamilton's equations (2.12); each S d (q k ) is seen as 
a function of the end point coordinates q k and the discrete end time k. Then, these action sums 
satisfy the discrete Hamilton- Jacobi equation (3.3). 



Proof. From Eq. (3.8), we have 



Ik+i) - SdiQk) — Pk+i ■ Qk+i - H^(q k ,p k+1 ), 
where Pk+i is considered to be a function of q k and qk+i, he-, Pk+i = Pk+i(Qk,Qk^ 



(3.9) 
Taking the 



derivative of both sides with respect to 

Pk+i + 



DS h d +1 ( qk+1 ) 



i, we have 
dpk+i 



[q k +i ~ D 2 H+(q k ,p k+1 )] . 



dqk+i 

However, the terms in the brackets vanish because the right discrete Hamilton's equations (2.12) 
are assumed to be satisfied. Thus, we have 



Pk+i 



DS k d +1 (q k+1 ). 



Substituting this into Eq. (3.9) gives Eq. (3.3). 



(3.10) 
□ 



Remark 3.4. Recall that, in the derivation of the continuous Hamilton- Jacobi equation [see, e.g., 
[T5l Section 23], we consider the variation of the action integral, Eq. (1.2), with respect to the end 
point (q, t) and find 

dS =pdq- H(q,p)dt. (3.11) 

This gives 

-H{q,p), p 



dt 

and hence the Hamilton- Jacobi equation 

dS 



as 

dq ' 



+ H 



as 

dq 



In the above derivation of the discrete Hamilton- Jacobi equation (3.3), the difference in two action 
sums, Eq. (3.9), is a natural discrete analogue of the variation dS in Eq. (3.11). Notice also that 
Eq. ( |3.9[ ) plays the same essential role as Eq. (3.11 ) does in deriving the Hamilton-Jacobi equation. 
Table [T] summarizes the correspondence between the ingredients in the continuous and discrete 
theories (see also Remark 3.4). 



3.3. The Right and Left Discrete Hamilton Jacobi Equations. Recall that, in Eq. (3.8), 
we wrote the action sum, Eq. (2.1), in terms of the right discrete Hamiltonian, Eq. (2.11). We can 
also write it in terms of the left discrete Hamiltonian, Eq. (2.14), as follows: 

fc-i 

SdiQk) = Y,i-P l ■ « " #5>J,<R+i)] • (3-12) 



1=0 



Then, we can proceed as in the proof of Theorem 3.3 First, we have 



Sd +1 (<7fc+i) - S$(qk) = -Pk -qk-H A (p k , q k 



+l 



(3.13) 
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Table 1. Correspondence between ingredients in continuous and discrete theories; 
R>o is the set of non- negative real numbers and No is the set of non- negative integers. 



Continuous 


Discrete 


{ q ,t)eQxR> 


(q k ,k) G Q x N 


q = dH/dp, 
p = —dH/dq 


q k+ i = D 2 H+(q k ,p k+1 ), 
p k = D 1 H^(q k ,p k+1 ) 


S(q,t) := f \p(s)-q(s)-H(q(s),p{s))]ds 
Jo 


k-1 

Sd(<7fc) := ^2[pi+i ■ m+i - H+(q u pi +1 )] 

1=0 


,„ dS , dS , 
db = —dq+ — dt 
oq at 


S* +1 (q k+1 )-S k d (q k ) 


pdq — H(q,p) dt 


Pk+i ■ qk+i - H^(q k ,p k+1 ) 


db / dS\ n 

m +H { q >8-q)=° 


S^ +1 (q k+ l) - S k d (q k ) - DS k+1 (q k+1 ) ■ q k+1 
+H+(q k7 DS k+1 (q k+1 )) =0 



where pk is considered to be a function of qk and qk+i, i-e., Pk = Pkiqk-, Qk+i)- Taking the derivative 
of both sides with respect to qk, we have 

-DS^(q k ) = -Pk ~ • [qk + D 1 H~ (p k , qk+i)} ■ 
oqk 



However, the terms in the brackets vanish because the left discrete Hamilton's equations (2.15) are 
assumed to be satisfied. Thus, we have 



p k = DS k d {q k ). 



(3.14) 



Substituting this into Eq. (3.13) gives the discrete Hamilton- Jacobi equation with the left discrete 
Hamiltonian: 



S k d +1 (q k+ i) - S^qk) + DS%(q k ) ■ q k + H~ (dS^), q k+ l) = 



(3.15) 



We refer to Eqs. (3.3) and (3.15) as the right and left discrete Hamilton- Jacobi equations, respec- 
tively. 



As mentioned above, Eqs. (3.8) and (3.12) are the same action sum, Eq. (2.1), expressed in 



different ways. Therefore we may summarize the above argument as follows: 



left discrete Hamilton- Jacobi equations, (3.3) and (3.15), respectively. 



Proposition 3.5. The action sums, Eq. (3.8) or equivalently Eq. (3.12) ; satisfy both the right and 



3.4. Explicit Forms of the Discrete Hamilton Jacobi Equations. The expressions for the 



right and left discrete Hamilton- Jacobi equations in Eqs. (3.3) and (3.15) are implicit in the sense 
that they contain two spatial variables qk and qk+i, Theorem 3.3 suggests that one may consider 



q k and q k +i to be related by the discrete Hamiltonian dynamics defined by either the right or 
left discrete Hamilton's equations (2.12) or (2.15), or equivalently, the discrete Hamiltonian map 
Fh d '■ (qk,Pk) (qk+i,Pk+i) defined in Eq. (2.8). More specifically, we may write qk+i m terms of 
qk- This results in explicit forms of the discrete Hamilton-Jacobi equations, and we shall define the 
discrete Hamilton-Jacobi equations by the resulting explicit forms. We will see later in Section [6] 
that the explicit form is compatible with the formulation of the Bellman equation. 
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For the right discrete Hamilton-Jacobi equation (3.3), we first define the map f k : Q 
follows: Replace Pk+i in Eq. (2.12a) by DS d +1 (q k+ \) as suggested by Eq. (3.10): 



Q as 



q k+1 = D 2 H+[q k ,DS k d +1 (q k+1 )) . 

Assuming this equation is solvable for q k+ i, we define f k '■ Q ^ Q by f k (q k ) = q k +i, i-e., f k is 
implicitly defined by 

f+(q k ) = D 2 H+(q k ,DS k+1 (f k + (q k ))) . (3.16) 

We may now identify q k +\ with f k (q k ) in the implicit form of the right Hamilton-Jacobi equa- 
tion (3.3): 



qk+lff+l 

d 



U + (<7)) " S k d (Q) ~ DS k d +\f+(q)) ■ f+(q)+H+(q,DS k d +Hti(<l))) = 0, (3.17) 
where we suppressed the subscript k of q k since it is now clear that q k is an independent variable 



as opposed to a function of the discrete time k. We define Eq. (3.17) to be the right discrete 



Hamilton-Jacobi equation. Notice that these are differential-difference-functional equations defined 
on Q x N, with the spatial variable q and the discrete time k. 



For the left discrete Hamilton-Jacobi equation (3.15), we define the map f k : Q — > Q as follows: 

f k (Qk) :=irQoF Ld (dS k d (q k ) 



(3.18) 



where ttq : T*Q — >• Q is the cotangent bundle projection; equivalently, f k is defined so that the 
diagram below commutes. 



T*Q 



T*Q 



dS k (q k ) h 



Q 



r. 



Q 



qk h 



F Ld (dS k (q k )) 



fk ilk 



Notice also that, since the map Fl a '■ (Qk,Pk) (lk+i,Pk+i) is defined by Eq. (2.15), A is defined 
implicitly by 

q k = -DtHifDSfatiJZiqkj) • (3.19) 



In other words, replace p k in Eq. (2.15a) by DS d (q k ) as suggested by Eq. (3.14), and define f k (q k ) 



as the q k+ i in the resulting equation. 

We may now identify q k +\ with f k (q k ) in Eq. (3.15): 



S k d +1 (f k (q)) ~ S k d (q) + DS« d (q) ■ q + H~ ( DS« d (q), f k (q) ) = 0, 



(3.20) 



where we again suppressed the subscript k of q k . We define Eqs. ( |3.17 ) and (3.20) to be the right 
and left discrete Hamilton-Jacobi equations, respectively. 



Remark 3.6. Notice that the right discrete Hamilton-Jacobi equation (3.17) is more complicated 
than the left one (3.20), particularly because the map f k appears more often than f k does in 



the latter; notice here that, as shown in Eq. (|3.18|), the maps f k in the discrete Hamilton-Jacobi 



equations (3.17) and (3.20) depend on the function S d , which is the unknown one has to solve for. 

However, it is possible to define an equally simple variant of the right discrete Hamilton-Jacobi 
equation by writing q k -\ in terms of q k : Let us first define g k : Q — > Q by 



9k{qk 



n Q oF^(dS k (q k ) 
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or so that the diagram below commutes. 



T*Q 



T*Q 



dS*(q k ) 



Q 



Hi: 



Q 



9k{qk) 



(ik 



Now, in Eq. (3.3), change the indices from (k, k + 1) to (k — 1, A:) and identify q k -\ with gk(q k ) to 
obtain 

Sfa) ~ S k d -\g k (q)) - DS k (q) ■ q + H+ (g k (q), DS k (q)) = 0, 
where we again suppressed the subscript k of q k . This is as simple as the left discrete Hamilton- 



Jacobi equation (3.20). However the map g k is, being backward in time, rather unnatural compared 



to f k . Furthermore, as we shall see in Section [6j in the discrete optimal control setting, the map f k 
is defined by a given function and thus the formulation with f k will turn out to be more convenient. 

3.5. The Discrete Hamilton Jacobi Equation on the Lagrangian Side. First, notice that 

(3.21) 



Eq. (2.1) gives 



S k d +1 (q k+1 )-S k d (q k 



Ld{qk,qk+i)- 



This is essentially the Lagrangian equivalent of the discrete Hamilton- Jacobi equation (3.17) as 



Lall and West [23] suggest. Let us apply the same argument as above to obtain the explicit form 



for Eq. (3.21). Taking the derivative of the above equation with respect to q k , we have 

-DiL d (q kl q k+ i)dq k = dS d (q k ), 
and hence from the definition of the left discrete Legendre transform, Eq. (2.6b), 

^L d (q k ,q k+1 ) = dS d {q k ). 
Assuming that FLJ is invertible, we have 

(5k,<fc+i) = (FLJ)" 1 ^*^)) = : (QkJt(Qk)), 
where we defined the map : Q — > Q as follows (see the commutative diagram below): 

ft(qk) :=pr 2 o(FL-)- 1 ( ( iS d fc ( (7fc ; 



(3.22) 



where pr 2 : Q x Q — > Q is the projection to the second factor, i.e., pJ^gi, (fe) = 92- Thus, eliminating 



q k +i from Eq. (3.21) and then replacing q k by q, we obtain the discrete Hamilton-Jacobi equation 



on the Lagrangian side: 



S k+1 (fk L (q))-S k d (q)=L d (q,f k L (q)). 



(3.23) 

The map f£ defined in Eq. (3.22) is identical to fj^ defined above in Eq. (3.18) as the commutative 
diagram below demonstrates. 



T*Q 



T*Q 



dS^q, 



{dS k (q k )) 
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The commutativity of the square in the diagram defines the /, as we saw earlier, whereas that 
of the right-angled triangle on the lower left defines the ft in Eq. (3.22); note the relation Fi d = 
FL+ o (FL")" 1 from Eq. ggfr . 

The map f£ being identical to fZ implies that the discrete Hamilton-Jacobi equations on the 
Hamiltonian and Lagrangian sides, Eqs. (3.20) and (3.23), are equivalent. 

4. Discrete Hamilton-Jacobi Theorem 

The following gives a discrete analogue of the geometric Hamilton-Jacobi theorem by Abraham 
and Marsden [1, Theorem 5.2.4]: 

Theorem 4.1 (Discrete Hamilton-Jacobi). Suppose that S d satisfies the right discrete Hamilton- 
Jacobi equation (3.17), and let {c k } k=Q C Q be a set of points such that 

c k+l = f+{c k ) for fe = 0,l,...,JV-l. 

Then, the set of points {(c k ,p k )} k=0 c T*Q with 

Pk ■= DS k d {c k ) 



(4.1) 



(4.2) 



is a solution of the right discrete Hamilton's equations (2.12). 

Similarly, suppose that S d satisfies the left discrete Hamilton-Jacobi equation (3.20), and let 
i c k} k= o C Q be a set of points that satisfy 

c k+1 = f k (c k ) for k = 0,l,...,N-l. (4.3) 

Furthermore, assume that the Jacobian Df k is invertible at each point c k . Then, the set of points 
{(ck,Pk)} k=0 C T*Q with 

p k := DS k (c k ) (4.4) 
is a solution of the left discrete Hamilton's equations ( |2.15 ). 



(4.5) 



Proof. To prove the first assertion, first recall the implicit definition of f k in Eq. (3.16): 

f k + {q)=D 2 H+( q ,DS k A +\f+{ q ))). 
In particular, for q = c k , we have 

Cfc+i = D 2 H+(c k ,p k ) , (4.6) 

where we used Eq. (4.1) and (4.2). On the other hand, taking the derivative of Eq. (3.17) with 
respect to q, 



DS k d +1 (f k + (q)) ■ Df+(q) - DS k d (q) - Df+(q) ■ D'S^V^q)) ■ f+(q) - DS^\f k + (q)) ■ Df+(q) 



2 ak+1. 



k+lft+i 



-,2 ck+1. 



which reduces to 



-DS k (q)+D 1 H^q,DS k+1 (f k + (Q))) =0, 



due to Eq. (4.5). Then, substituting q = c k gives 

-DS k (c k ) + DtH+fc^DS^Hf+ick)) ) = 0. 



Using Eqs. (4.1) and (4.2), we obtain 



Pk = DiHf(c k ,p k 



+1 



(4.7) 



Eqs. (4.6) and (4.7) show that the sequence {(c k ,p k )} satisfies the right discrete Hamilton's equa- 
tions |2T2j) . 
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Now, let us prove the latter assertion. First, recall the implicit definition of /. in Eq. (3.19): 

q = -D 1 H d (DS k d (q),f k (q)) (4.8) 

In particular, for q = c k , we have 

c k = -DiH~(p k ,c k+1 ) , (4.9) 



where we used Eq. (4.3) and (4.4). On the other hand, taking the derivative of Eq. (3.17) with 
respect to q yields, 

DS k +\f k -(q)) ■ Df k {q) - DS k (q) + D 2 S k (q) ■ q + DS k {q) 

+ (DS k (q), Kb)) ■ D 2 S k (q) + D 2 H d (DS k (q),f k (q)) ■ Df k (q) = 0, 



Df k (q) = 0, 



which reduces to 

~DS k+1 (f k (q)) + D 2 H d (DS k (q),f k (q) 
due to Eq. (4.8). Then, substituting q = c k gives 

DS k+1 (f k (c k )) = -D 2 H d (DS k (c k ),f k (c k )) , 



since Df, (c k ) is invertible by assumption. Then, using Eqs. (4.3) and (4.4), we obtain 

Pk+i = -D 2 H d (p k ,c k+1 ) . 



(4.10) 



Eqs. (4.9) and (4.10) show that the sequence {(c k ,p k )} satisfies the left discrete Hamilton's equa- 
tions |Tl5]) . □ 

5. Application To Discrete Linear Hamiltonian Systems 

5.1. Discrete Linear Hamiltonian Systems and Matrix Riccati Equation. 

Example 5.1 (Quadratic discrete Hamiltonian — discrete linear Hamiltonian systems). Consider a 
discrete Hamiltonian system on T*W n = M n x W 1 (the configuration space is Q = W 1 ) defined by 
the quadratic left discrete Hamiltonian 



H d (p k ,q k+1 ) = ^plM 1 p k +plLq k+ i + -ql +1 Kq k+ i, 



(5.1) 



where M , K, and L are real n x n matrices; we assume that M and L are invertible and also that 
M and K are symmetric. The left discrete Hamilton's equations (2.15) are 

q k = -(M _1 p fc + Lq k+1 ), 

Pk+i = ~{L T Pk + Kq k+ i), 



or 



/ Qh+l\ 




-L-Hl- 1 
KL-Hr 1 - L T 




(5.2) 



and hence are a discrete linear Hamiltonian system (see Section A.l). 



Now, let us solve the left discrete Hamilton-Jacobi equation (3.20) for this system. For that 
purpose, we first generalize the problem to that with a set of initial points instead of a single initial 
point (qo,po). Mor e spe cifically, consider the set of initial points that is a Lagrangian affine space 
C Zo (see Definition A. 2) which contains the point zq := (qo,po). Then, the dynamics is formally 
written as, for any discrete time k G N, 

#*) := { p Ld f (£) = F Ld o • • • o F Ld (C ZQ ) , 
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where Fi d : T*Q — > T*Q is the discrete Hamiltonian map defined in Eq. (2.8). Since Fi d is a 
symplectic map, Proposition A. 4 implies that is a Lagrangian affine space. Then, assuming 
that is transversal to {0} © Q* , Corollary A. 6 implies that there exists a set of functions S 1 ^ of 
the form 

S k d {q) = \q T A k q + b T k q + c k: (5.3) 

such that C,^ = graph dS^; here A k are symmetric n x n matrices, b k are elements in M. n , and c k 
are in M. 

Now that we know the form of the solution, we substitute the above expression into the discrete 
Hamilton-Jacobi equation to find the equations for A k , b k , and c k . Notice first that the map f7 is 
given by the first half of Eq. \h.2\ with p k replaced by DS^ (q) : 



A" (9) 



-L- l [q + M~ l DS k d {q) 
-L~ l {I + M- l A k )q- L^M-' h 



k- 



(5.4) 



Then, substituting Eq. (5.3) into the left-hand side of the left discrete Hamilton-Jacobi equa- 

id 



tion (3.20) yields the following recurrence relations for A k , b k , and c k : 

A k +\ = L T (I + AfcM _1 )~ 1 AfcL 
b k+1 = -L T (I + A k M~ 1 )- 1 b k , 



Cfc+l 



where we assumed that / + A k M 1 is invertible 



Ck 



\b T k (M + A k )~H k , 



(5.5a) 
(5.5b) 

(5.5c) 



Remark 5.2. For the A k+ \ defined by Eq. (5.5a) to be symmetric, it is sufficient that A k is invertible; 



for if it is, then Eq. (5.5a) becomes 

A k+l 



L 1 (AZ + M~ )~ L — K, 
and so A k , M, and K being symmetric implies that A k+ \ is as well. 



Remark 5.3. We can rewrite Eq. (5.5a) as follows: 



A k+1 = [KL- 1 + (KL~ l M~ l - L T )A k ] (-L' 1 - L" 1 M~ l Ah)" 1 . (5.6) 
Notice the exact correspondence between the coefficients in the above equation and the matrix 



entries in the discrete linear Hamiltonian equations (5.2). In fact, this is the discrete Riccati 



equation that corresponds to the iteration defined by Eq. (5.2). See Ammar and Martin [2] for 
details on this correspondence. 

To summarize the above observation, we have: 



Proposition 5.4. The discrete Hamilton-Jacobi equation (3.20) applied to the discrete linear 



Hamiltonian system (5.2) yields the discrete Riccati equation (5.6). 



In other words, the discrete Hamilton-Jacobi equation is a nonlinear generalization of the discrete 
Riccati equation. 

6. Relation to the Bellman Equation 
In this section, we apply the above results to the optimal control setting. We will show that the 



(right) discrete Hamilton-Jacobi equation (3.17) gives the Bellman equation (discrete-time HJB 
equation) as a special case. This result gives a discrete analogue of the relationship between the 



H-J and HJB equations discussed in Section 1.3 
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6.1. Discrete Optimal Control Problem. Let qd '■= {qk} k= o De t ne state variables in a vector 



space V = M. n with q$ and fixed and Ud '■= {uk} k= o ^ e controls in the set U C W 71 . With 
given function Cd ■ V x U — > R, define the discrete cost functional 

jV-1 



a 



Then, we formulate the Standard Discrete Optimal Control Problem as follows [see, e.g..l4"ll9|U7M21j: 

Problem 6.1 (Standard Discrete Optimal Control Problem). Minimize the discrete cost functional, 
i.e., 

N-l 

minJ d = minV' C d (q k ,u k ), (6.1) 

u d u d ^— ' 

k=0 

subject to the constraint, 

Qk+l = fd{qk,Uk)- (6.2) 

6.2. Necessary Condition for Optimality and the Bellman Equation. We would like to 
formulate the necessary condition for optimality. First, introduce the augmented discrete cost 
functional: 

N-l 



Sd(qd,Pd,u d ) := ^2{Cd{q k ,u k ) + p k +i - [qk+l ~ fd(qk,u k )}} 

k=0 
N-l 

= ^2 [Pk+i ■ Qk+i - H+(q k ,p k+1 ,u k ) 



k=0 

where we introduced the costate pd '■= {pk} k= \ with p k 6 V*, and also defined the discrete control 
Hamiltonian 

Ht(q k ,Pk+i,u k ) ■= p k +i ■ fd(qk,Uk) ~ Cd(qk,u k )- (6.3) 



Then, the optimality condition, Eq. (6.1), is restated as 

N-l 

Sd(qd,Pd,u d ) = min 

9d.iM.Ud 



JV— X 

min S d (qd,Pd,Ud) = min V" \Pk+l • Qk+l ~ Hf(q k ,p k+ i, u k ) 
'd>Pd.Wd gd,Pd,«d ^ L 



k=o 

In particular, extremality with respect to the control u d implies 

D 3 H+(q k , Pk+1 ,u k ) = 0, fc = 0,1,..., JV-1. (6.4) 

Now, we assume that is sufficiently regular so that this equation uniquely determines the 
optimal control u d := {u* k } k ~^; and therefore, u* k is a function of q k and p k +\, i.e., u* k = u k (q k ,p k+ %). 
We then define 

Hf(q k ,p k+ i) := max (q k ,p k+1 ,u k ) 

= max[p fc+ i • f d (q k ,u k ) - C d (q k ,u k )} 



Pk+i ■ fd(qk, u* k ) - C d (q k , u* k ), (6.5) 
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and also the optimal discrete cost-to-go function 

N-l 



4( 



'■= Yl { C dfa> u l) + »+l • - /d(«, «*)]} 



Z=fc 

AT-l 



l=k 

where S d is the optimal discrete cost functional, i.e., 

N-l 

Sd '■= Sd(qd,Pd,u* d ) = 

k=0 

and 



(6.6) 



Y [Pk+l ■ Qk+i ~ H^{q k ,p k+1 )] . 



S k d(qk) 



k-l 

E 

1=0 



[Pl+i ■ Ql+i ~ H^(qi,pi +1 )} 



The above action sum has exactly the same form as Eq. (3.8) formulated in the framework of 
discrete Hamiltonian mechanics. Therefore, our theory now directly applies to this case: The 
corresponding right discrete Hamilton's equations (2.12) are, using the expression in Eq. (6.5), 

Qk+i = fd(qk,u%), 

Pk =Pk+i ■ Dif A (q k ,u* k ) - DiC A {q k ,u* k ). 



Therefore, Eq. (3.16) gives the implicit definition of f^ as follows: 

4 + (%) = u(q k ,ut(q k ,DS k d +1 (f+(q k )))) . 
Hence, the (right) discrete Hamilton-Jacobi equation (3.17) applied to this case gives 



(6.7) 



ok+l, 

^d 



(/d (?*,<)) - S k d (q k ) - DS k d +1 (f d (q k ,u* k )) ■ f d (q k ,u* k ) + H+ (q k , DS k +\f d (q k , <))) = 0, 



and again using the expression for the Hamiltonian in Eq. (6.5), this becomes 

max S d +1 (f d (q k ,u k )) - C d (q k ,u k ) = S d (q k ). 

u k L J 

Since S d (q k ) = S d - J d (q k ), we obtain 



mm 



J; 



k+li 



4(Qk) 



(6.8) 



>d (fd(qk,u k )) + C d (q k ,u k ) 
which is the Bellman equation (see, e.g., Bellman [HE] and Bertsekas [6]). 

Remark 6.2. Notice that the discrete HJB equation ( |6.8[ ) is much simpler than the discrete Hamilton- 
Jacobi equations (3.17) and (3.20) because of the special form of the control Hamiltonian Eq. (6.5). 
Also, notice that, as shown in Eq. (6.7), the term fu(q k ) is written in terms of the given function 
/. See Remark 3.6 for comparison. 

6.3. Relation between the Discrete H —J and Bellman Equations and its Consequences. 

Summarizing the observation made above, we have 



Proposition 6.3. The right discrete Hamilton-Jacobi equati on (|3.17 ) applied to the Hamiltonian 
formulation of the Standard Discrete Optimal Control Problem 6.1 gives the Bellman equation (6.8). 

This observation leads to the following well-known fact: 
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Proposition 6.4. Let J d {qk) be a solution to the Bellman equation (6.8). Then, the costate pk in 
the discrete maximum principle is given as follows: 



where c k -\ 



p k = -DJZ(c k ), 
fd(ck,ut) with the optimal control ut. 



Proof. Follows from a reinterpretation of Theorem 4.1 through Proposition 6.3 with the relation 
S k d (q k ) = S* d - 4(q k ). □ 



7. Generalized Bellman Equation with Internal- Stage Controls 

In the previous section, we showed that the discrete Hamilton-Jacobi equation recovers the 
Bellman equation if we apply our theory to the Hamiltonian formulation of the Standard Discrete 



Optimal Control Problem 6.1 In this section, we generalize the approach to derive what may be 



considered as higher-order discrete-time approximations of the HJB equation (1.4). Namely, we 



derive a class of discrete control Hamiltonians that use higher-order approximations (a more general 



version of Eq. (6.3)) by employing the technique of Galerkin Hamiltonian variational integrators 
introduced by Leok and Zhang [26]; and then, we apply our theory to obtain a class of generalized 
Bellman equations that have controls at internal stages. 

7.1. Continuous-Time Optimal Control Problem. Let us first briefly review the standard 
formulation of continuous-time optimal control problems. Let q be the state variable in a vector 
space V = M n , qo and qT fixed in V, and u be the control in the set IA C M. m . With a given function 
C : V x U — > M, define the cost functional 

J:= f C(q(t),u(t))dt. 
Jo 

Then, we formulate the Standard Continuous- Time Optimal Control Problem as follows: 

Problem 7.1 (Standard Continuous-Time Optimal Control Problem). Minimize the cost func- 
tional, i.e., 

f T 

minJ = min / C(q(t),u(t)) dt, 
«(•) «(•) Jo 

subject to the constraints, 

q = f(q,u), 

and q(0) = qo and q(T) = qr- 

A Hamiltonian structure comes into play with the introduction of the augmented cost functional: 

S := [ T {C(q(t),u(t)) + p(t)[q(t) - f(q(t),u(t))}} dt 
J 
T 

' dt, 







p(t)q(t)-H{q(t),p(t),u(t)) 

where we introduced the costate p{t) G V* , and also defined the control Hamiltonian, 

H(q,p,u):=p- f(q,u)-C(q,u). (7.1) 
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7.2. Galerkin Hamiltonian Variational Integrator. Recall, from Leok and Zhang [26, Sec- 
tion 2.2], that the exact right discrete Hamiltonian is a type-two generating function for the original 
continuous-time Hamiltonian flow, defined by 

H+ ex (q , Pl ) = ext \ Pl qi - [ \p(t)q(t) - H(q(t),p(t))] dt) , (7.2) 

(q, P )eC 1 ([o,h],T*Q) I Jo J 



q{0)=qo,P(h)=p 



where h is the time step; C 1 ([0, h],T*Q) is the set of continuously differentiable curves on T*Q over 
the time interval [0, h]; an extremum is achieved for the exact solution of Hamilton's equations ( |1.1| ) 
that satisfy the specified boundary conditions. Therefore, it requires the exact solution (q(t),p(t)) 
to evaluate the the above integral, and so the exact discrete Hamiltonian cannot be practically 
computed in general. 

The key idea of Galerkin Hamiltonian variational integrators [26] is to replace the set of curves 
C 1 ([0, h], T*Q) by a certain finite-dimensional space so as to obtain a computable expression for a 
discrete Hamiltonian. 

7.3. Galerkin Discrete Control Hamiltonian. Here, we would like to apply the above idea to 
the control Hamiltonian, Eq. (7.1), to obtain a discrete control Hamiltonian. 
Let Q(V) be a finite-dimensional space of curves defined by 

C s d {V) := ^t^Y^w^iit/h) | t G [0,h], w i € V for each i € {1, 2, . . . , s}| . 

with the basis functions {ipi : [0, 1] — > M}| =1 . 

1. Use the basis functions ipi to approximate the velocity q over the interval [0, h], 

s 

q{rh) « q d (Th) = ^ioVi(r), 
i=i 

where r € [0, 1] and w l G V for each %= 1, . . . , s. 

2. Integrate qd(t) over [0,r/i], to obtain the approximation for the position q, i.e., 

q d (rh) = q d (0) + / S^w^^t/h) dt = q + hY^w* i>i{p)dp, 
Jo i=1 i=1 Jo 

where we applied the boundary condition ^(0) = qo. Applying the boundary condition qd(h) = 
qi at the other endpoint yields 

fl s 



qi = q d {h) = q + h^w 1 / ipi(p) dp = q + B i w \ 
i=i J° i=i 

where Bi := L ipi(r) dr. Furthermore, we introduce the internal stages, 

S „ c i S 

Q\w) := q d {c l h) = qo + hS2w j / ^-(r) dr = q + h V A)w j , (7.3) 
where A 1 - := J Q C ipj(r)dT. 



3. The exact discrete control Hamiltonian H~t is defined as in Eq. (7.2): 



d,ex 

h 



HJ ex (q ,Pi;u(-)) := ext \piqi~ p(t)q(t) - H(q(t),p(t),u(t)) 

hrf»)eC 1 ([o,fc],VxV*) I Jo L 

?(to)=90,p(tl)=Pl 



dt 



DISCRETE HAMILTON-JACOBI THEORY 



19 



Again this is practically not computable, and so we employ the following approximation: Use 
the numerical quadrature formula 



J ° i=i 



with constants (pi, c l ) and the finite-dimensional function space C|(V) to construct H^(qo,Pi, U) 
as follows: 



H+(q ,Pi,U) ■= ext l Pl q d (h) - h Y> Uc'h^h) - H{Q i {w),P\U i ) 
= extK(q ,w,P,U,pi), 

W.P 

where we set P l := p{c l h) and U l := u(c l h) and defined 

K(q ,w,P,U, Pl ) :=pi • g + h^B^ - h^b 

V i=l / i=l 

= Pi • I go + /i -B, 



j'=i 



i=l 



i=i 



where we defined Mj := ipjic 1 ) and used the expression for the control Hamiltonian in Eq. (7.1 ); 
note that P l £ V* and w' 1 G V for each i = 1, . . . , s, and that / takes values in V. In or- 
der to obtain an expression for Ht (go , Pi > U) , we first compute the stationarity conditions for 
K(qo,w, P,U, P i) under the fixed boundary condition (qo,pi): 



- dK ^o,w,P,U, Pl ) = ^ . B ._ h J2 b . [ M i P * _ hA i Dl Htf( w)jI » } W) 

i=i 



dw 3 

dK(q ,w,P,U,pi) 



dP l 



-hbi 



^M)^-/(Q>),C/ i ) 
j'=i 



(7.4a) 
(7.4b) 



for j = 1, . . . , s. 



4. By solving the 2s stationarity conditions ( |7.4| , we can express the parameters w and P in terms 
of go, pi, and U, i.e., it; = w(go> U) and P = P(go,Pi, J7): In particular, assuming 6j ^ for each 
j = 1, . . . , s, Eq. ( |7.4b ) gives w^M % - = f(Q l (w), U i ); this gives a set of ns nonlinear equations ^1 
satisfied by w = w(qo, U)^ Therefore, we have 



K(q ,w(q ,U),P,U,pi) = pi 



q + h^2B j w j (q ,U) 

i=l 



hY,b i C(Q i (w(qo,U)),U i ) 



8=1 



2 Recall that ui J £ V and / takes values in V. 

^Note from Eq. (7.31 that Q' is written in terms of qo and w. 
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Notice that the internal-stage momenta, P l , disappear when we substitute w = w(qo, U). There- 
fore, we obtain the following Galerkin discrete control Hamiltonian: 



(7.5) 



H£(qo,pi,U) := K[q ,w(q Q ,U),P(q ,pi,U) : U,p 1 

h^biCiQ'iw^U)),^) 



Pi ■ 



q + h^2B j w j (q ,U) 



i=l 



7.4. The Bellman Equation with Internal-Stage Controls. The Galerkin discrete control 
Hamiltonian, Eq. (7.5), gives 

H£(q k ,Pk+l, Ul, Uf.) := p k+1 ■ f d (q k , C/jJ, ...,£/£)- C d (q k , C/jJ, [/£), 

with 

s 

f d (q k , Ul..., U s k ) := qk + hY, Ul . . • , U s k ) B u 

and 



i=l 



C d (q k , Ul,..., U s k ) -hj^bi CiQ'iwiQk, U l k , . . . , Ufi), W k ) 



(7.6) 
(7.7) 

(7.8) 



i=i 



This is a generalized version of Eq . (|6.3[ ) with internal-stage controls {J7|}| =1 as opposed to a single 
control u k per time step (see Fig. I2l). Now assume that 



_d_ 

m 



H^(q k ,Pk+i,Ul,...,U k ) = 0, i = l,...,s, 



(7.9) 



is solvable for {U k } s i=1 to give the optimal internal-stage controls {U k ' l }f =1 . Then, we may apply the 
same argument as in Section [6j In particular, the right discrete Hamilton- J acobi equation (3.17) 
applied to this case gives the following Bellman equation with internal- stage controls: 



mm 

u 1 II s 



J k d +1 (h(q k , Ul..., U s k )) + C d (q k , Ul,..., U s k )] = J k A (q k ). 



(7.10) 



q k 



— i 1 1 — 

t t t 

ul ul ul 



t 

Ul 



Figure 2. Internal-stage controls {Ul}f =l in the discrete time intervals between k 
and k + 1. 



The following example shows that the standard Bellman equation (6.8) follows as a special case: 



Example 7.2 (The standard Bellman equation). Let s = 1, and select 

*i(r) = l, h = l, c l = 0. 

Then, we have Si = 1,A\ = 0, and M \ = 1 . Hence, Eq. (1731) gives Q 1 = q k (we set the endpoints 
(q , qi ) to be (q k ,q k+ i) here), and Eq. ( |7.4b[ ) gives 

w x = f{q k ,Ul). 

However, the control Ul is defined as follows (we shift the time intervals from [0, h] to [t k ,t k + h] 
here) : 

Ul := u(t k + c l h) = u(t k ) =: u k . 
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So, we have w 1 = f(q k ,u k ), and thus, Eqs. (7.7) and (|7,8|) give 



and 



fd(Qk,Uk) = Qk + hf(q k ,u k ), 



C d (q k ,u k ) = hC(q k ,u k ), 

respectively. Notice that this approximation gives the forward-Euler discretization of the Standard 
Continuous-Time Optimal Control Problem 7.1 to yield the Standard Discrete Optimal Control 
Problem 6.1 In fact, the Bellman equation with internal-stage controls, Eq. (7.10), reduces to the 
standard Bellman equation (6.8): 



mm 



Jd +1 (fd(Qk, u k )) + C d (q k , u k ) = J d {q k ). 



(7.11) 



Remark 7.3. Higher-order approximations with any number of s are possible as long as Eq. ( 7.4b| ) 
is solvable for w. See Leok and Zhang [26] for various different choices of discretizations. 



7.5. Application to the Heisenberg System. Let us now apply the above results to a simple 
optimal control problem to illustrate the result: 

Example 7.4 (The Heisenberg system; see, e.g., Brockett [8] and Bloch [?]). Consider the following 
optimal control problem: For a fixed time T > 0, 



mm 



u (')> v (') Jo 2 



1 



{vT + v l ) dt, 



subject to the constraint, 



x = u, 



y = v, 



z = uy — vx. 



This is the Standard Continuous-Time Optimal Control Problem 7.1 with V = M 3 , U 

1 = (x,y,z), and 



f(x,y,z,u,v) 




C(x,y,z,u,v) = \(u 2 + v 2 ) 



If we apply the choice of the discretization in Example 7.2 we have the standard Bellman equation 



mm 

Uk,Vk 



S d (fd{qk,Uk,Vk)) - C d (q k ,u k ,v k ) -S d (q k ) = 0, 



with q k := (x k ,y k , z k ), where 
fd(qk,Uk,Vk) = 



x k + hu k 
Vk + hv k 
Zk + h{u k y k - v k x k ) 

Now, if we choose s = 2, and select 



(*i(t), * 2 (t)) = (1, cos(vrr)), b = (b 1} b 2 
Then, we have 

B = (B 1 ,B 2 ) = (1,0), A. 



C d (q k ,u k ,v k ) 



1 1 

2' 2 



oO* + V k)- 



(c\c 2 ) = (0,1). 





1 



M 



1 1 
1 -1 



Leok and Zhang |26| Example 4.4] show that this choice of discretization corresponds to the 
Stormer-Verlet method (see, e.g., Marsden and West [30]). The Bellman equation with internal- 
stage controls, Eq. (7.10), then becomes 



mm 

M k' v k' u k'"t 



J t +l {fd{qk,u\,vl,ul,vl)) + C d (q k ,u\,vl,ul,vl) = J d {q k ), 
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where 



fd[qk,ul,vl,ul,vl) 



x k + h {u\ + u\)/2 
y k + h{vl + v 2 k )/2 



z k + h 



Vk 



and 



Cd{qk,ul,vl,u 2 k ,v 2 k ) 



v l + v l , u l v l ~ u l v l 
~2 k 4 



,2\2n 



K) 2 + K) 2 , K) 2 + K) 



+ 



8. Conclusion and Future Work 

We developed a discrete-time analogue of Hamilton-Jacobi theory starting from the discrete 
variational Hamiltonian mechanics formulated by Lall and West [23]. We reinterpreted and ex- 
tended the discrete Hamilton-Jacobi equation given by Elnatanov and Schiff [13J in the language of 
discrete mechanics. Furthermore, we showed that the discrete Hamilton-Jacobi equation reduces to 
the discrete Riccati equation with a quadratic Hamiltonian, and also that it specializes to the Bell- 
man equation of dynamic programming if applied to standard discrete optimal control problems. 
These results are discrete analogues of the corresponding known results in the continuous-time 
theory. Application to discrete optimal control also revealed that the Discrete Hamilton-Jacobi 
Theorem 4.1 specializes to a well-known result in discrete optimal control theory. We also used 



a Galerkin-type approximation to derive Galerkin discrete control Hamiltonians. This technique 
gave an explicit formula for discrete control Hamiltonians in terms of the constructs in the original 
continuous-time optimal control problem. By viewing the Bellman equation as a special case of 
the discrete Hamilton-Jacobi equation, we could introduce the discretization technique for discrete 
Hamiltonian mechanics into the discrete optimal control setting; this lead us to a class of Bellman 
equations with controls at internal stages. 

We are interested in the following topics for future work: 

• Application to integrable discrete systems: Theorem 4.1 gives a discrete analogue of the 



theory behind the technique of solution by separation of variables, i.e., the theorem relates 
a solution of the discrete Hamilton-Jacobi equations with that of the discrete Hamilton's 
equations. An interesting question then is whether or not separation of variables applies to 
integrable discrete systems, e.g., discrete rigid bodies of Moser and Veselov [3T] and various 
others discussed by Suris [3_3l [M] . 

Development of numerical methods based on the discrete Hamilton-Jacobi equation: Hamilton- 
Jacobi equation has been used to develop structured integrators for Hamiltonian systems. 
Ge and Marsden p3] developed a numerical method that preserves momentum maps and 
Poisson brackets of Lie-Poisson systems by solving the Lie-Poisson Hamilton-Jacobi equa- 
tion approximately. See also Channell and Scovel [11] (and references therein) for a survey 
of structured integrators based on the Hamilton-Jacobi equation. The present theory, being 
inherently discrete in time, potentially provides a variant of such numerical methods. 

Extension to discrete nonholonomic and Dirac mechanics: The present work is concerned 
only with unconstrained systems. Extensions to nonholonomic and Dirac mechanics, more 
specifically discrete-time versions of the nonholonomic Hamilton-Jacobi theory [lUt [T2"| flU] 
[32] and Dirac Hamilton-Jacobi theory |27| . are another direction for future research. 

Relation to the power method and iterations on the Grassmannian manifold: Ammar and 
Martin [2] established links between the power method, iterations on the Grassmannian 
manifold, and the Riccati equation. The discussion on iterations of Lagrangian subspaces 



and its relation to the Riccati equation in Sections 5.1 and A. 2 is a special case of such links. 
On the other hand, Proposition 5.4 suggests that the discrete Hamilton-Jacobi equation 
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is a generalization of the Riccati equation. We are interested in exploring possible further 
links implied by the generalization. 
• Galerkin discrete optimal control problems: The Galerkin discrete control Hamiltonians 
may be considered to be a means of formulating discrete optimal control problems with 
higher-order of approximation to a continuous-time optimal control problem. This idea 
generalizes the Runge-Kutta discretizations of optimal control problems (see, e.g., Hager 
|18| and references therein). In fact, Leok and Zhang |26j showed that their method recovers 
the SPRK (symplectic-partitioned Runge-Kutta) method. Therefore, this approach is ex- 
pected to provide structure-preserving higher-order numerical methods for optimal control 
problems. 
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Appendix A. Discrete Linear Hamiltonian Systems 

A.l. Discrete Linear Hamiltonian Systems. Suppose that the configuration space Q is an n- 
dimensional vector space, and that the discrete Hamiltonian or is quadratic as in Eq. ( |5.1[ ). 
Also assume that the corresponding discrete Hamiltonian map Fi d : (qk,Pk) ^ (<lk+i,Pk+i) is 



invertible. Then, the discrete Hamilton's equations (2.12) or (2.15) reduce to the discrete linear 
Hamiltonian system 

z k+ i = A Ld z k , (A.l) 
where z k £ M 2n is a coordinate expression for (qk,Pk) £ Q © Q* and A^ d : Q © Q* — > Q © Q* is 
the matrix representation of the map Fl a under the same basis. Since Fi d is symplectic, Ai d is an 
2n x 2n symplectic matrix, i.e., 

AljA Ld = I (A.2) 

where the matrix J is defined by 



/ 
-I 



with / the n x n identity matrix. 



A.2. Lagrangian Subspaces and Lagrangian Affine Spaces. Let us recall the definition of a 
Lagrangian subspace: 

Definition A.l. Let V be a symplectic vector space with the symplectic form Q. A subspace £ 
of V is said to be Lagrangian if £l(v, w) = for any v,w G C and dim£ = dim V/2. 

We introduce the following definition for later convenience: 

Definition A.2. A subset Cb of a symplectic vector space V is called a Lagrangian affine space if 
Cb = b + C for some element b € V and a Lagrangian subspace £ C V. 

The following fact is well-known (see, e.g., Jurdjevic |22[ Theorem 6 on p. 417]): 

Proposition A. 3. Let £ be a Lagrangian subspace of V and $ : V — > V be a symplectic transfor- 
mation. Then, for any k £ N, the image of £ under the k-fold composition of i.e., 

§ k {£) = $o---o3> (£) 

k 

is also a Lagrangian subspace of V . 

A similar result holds for Lagrangian affine spaces: 
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Proposition A. 4. Let Cb = b+C be a Lagrangian affine space ofV and : V — > V be a symplectic 
transformation. Then ^ k (Cb) is also a Lagrangian affine space ofV for any k 6 N. More explicitly, 
we have 

= $ k b + <Z> k (C). 

Proof. Follows from a straightforward calculation. □ 

A. 3. Generating Functions. Now, consider the case where V = QffiQ* to apply the results from 
Section |A.2| to the setting in Section A.l This is a symplectic vector space with the symplectic 



form Vt:{Q®Q*)x{Q® Q*) -> R defined by 

Q. : (v, w) i Y v T Jw. 

The key result here regarding Lagrangian subspaces on Q © Q* is the following: 

Proposition A. 5. A Lagrangian subspace of Q © Q* that is transversal to {0} © Q* is the graph 
of an exact one-form on Q, i.e., C = graph alS for some function S : Q — > R which has the form 

S{q) = \(Aq,q)+C (A.3) 

with some symmetric linear map A : Q —> Q* and an arbitrary real scalar constant C . Moreover, the 
correspondence between the Lagrangian subspaces and such functions (modulo the constant term) 
is one-to-one. 

Proof. First, recall that a Lagrangian submanifold of T*Q that projects diffeomorphically onto Q 
is the graph of a closed one-forms on Q (see Abraham and Marsden [H Proposition 5.3.15 and the 
subsequent paragraph on p. 410]). In our case, Q is a vector space, and so the cotangent bundle 
T*Q is identified with the direct sum Q © Q* . Now, a Lagrangian subspace of Q © Q* that is 
transversal to {0} © Q* projects diffeomorphically onto Q, and so is the graph of a closed one-form. 
Then, by the Poincare lemma, it follows that any such Lagrangian subspace C is identified with 
the graph of an exact one-form dS with some function S on Q, i.e., £ = graph dS. 

However, as shown in, e.g., Jurdjevic [221 Theorem 3 on p. 233], the space of Lagrangian subspaces 
that are transversal to {0} © Q* is in one-to-one correspondence with the space of all symmetric 
maps A : Q — > Q* , with the correspondence given by C = graph A. Hence, graph dS = graph A, or 
more specifically, 

dS(q) = A tJ ql dq\ 

This implies that S has the form 

1 

2" 

with an arbitrary real scalar constant C. □ 



S(q) = -A ij q i qi + C, 



Corollary A. 6. Let C Zo = zq + C be a Lagrangian affine space, where zq = (</o>Po) is an element 
in Q © Q* and £ is a Lagrangian subspace of Q © Q* that is transversal to {0} © Q* . Then, C Zo is 
the graph of an exact one-form dS with a function S : Q — ?■ K of the form 

S(q) = ^ (M, q) + (Po - Aq Q , q) + C, (A.4) 
with some symmetric linear map A : Q — >■ Q* and an arbitrary real scalar constant C . 



Proof. From Proposition A. 5 there exists a function S : Q — > R of the form Eq. (A.3) with some 
symmetric linear map A : Q — > Q* such that £ = graph dS = graph A. Let us define S : Q — > R by 

S{q) := S{q- q ) + (p ,q) , 



from which Eq. (A.4) follows by direct calculation. Then, 

dS(q) = A(q - q ) + p . 
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and thus 

gv a phdS={(q,dS(q)) \q£Q} 

= {{q, Mq - go) + po) \q^Q} 

= {qo,po) + {(q-qo,Mq-Qo)) I q e Q} 

= z + graph A 
= z + C 
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